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Abstract : 

In order to process Space Shuttle 
vehicles for launch, the various Shuttle 
systems are subjected to various test 
and checkout procedures prior to launch. 
The system of interest in this paper is 
the Shuttle Data Processing System (DPS) 
and, in particular, the DPS Multi- 
Function CRT Display System (MCDS) . Due 
to the complexity of the Shuttle as a 
whole and DPS in particular, the system 
may at times behave in an unpredictable 
yet benign manner in respect to normal 
operations. Therefore, it is difficult 
for even experienced systems engineers 
to determine whether an annunciated er- 
ror is truly a failure or a benign 
anomaly. An automated, prototype diag- 
nostic tool is to be described in order 
to provide a solution to the labor in- 
tensive and time consuming diagnostic 
techniques currently used. The MCDS 
Diagnostic Tool (MDT) will be capable of 
monitoring the MCDS system real time, 
recognizing and analyzing failures 
giving the user a probable cause of the 

failure. The MDT is considered to be a 
pioneering diagnostic system for all DPS 
subsystems diagnostics. 

The primary goal at the Kennedy Space 
Center is to prepare the space Shuttle 
system, both the Shuttle and its 
payload, for launch into low earth or- 
bit. In order for this goal to be ac- 
complished in an efficient yet safe man- 
ner, all Shuttle sub-systems are sub- 
jected to various test and checkout pro- 
cedures prior to launch. A majority of 
these procedures are carried out by sys- 
tems engineers (via ground software) 
from firing room resident computer con- 
soles. This network of computer con- 
soles which constitutes a main component 
of the ground Launch Processing System 
(LPS) , is connected to the Shuttle via a 
Launch Data Bus (LDB) . In this manner, 
systems engineers are able to monitor 
and control vehicle subsystems whether 


the Shuttle is residing in the orbiter 
processing facility, vehicle assembly 
building or pads . 

The Shuttle system of interest in this 
paper is the Shuttle' s Data Processing 
System (DPS) . The DPS is composed of 
(1) General Purpose Computers (GPC) (2) 
Multi-Function CRT Display System (MCDS) 
(3) Mass Memory Units (MMU) , and (4) 
Multiplexer/Demultiplexer (MDM) and re- 
lated software. As is currently done, 
checkout and configuration of DPS sub- 
systems are done by a DPS systems en- 
gineer initiating and controlling a set 
of ground software programs. Additional 
system configurations are done by manual 
switch settings inside the cockpit via 
voice instruction to a space craft 
operator . 

In order to ensure the correct function- 
ing of Shuttle systems, some level of 
automatic error detection has been in- 
corporated into all Shuttle systems. 
For the DPS system, error detection 
equipment has been incorporated into all 
its subsystems. This error detection 
equipment is typically manifested as 
electronic circuitry composed of 
hardware registers where the bits of a 
particular register corresponds to par- 
ticular errors (i.e. power transient 
detected) . Additional errors are annun- 
ciated using both visual cues (i.e. 
mechanical flags and lights) and 
auditory cues (i.e. alarms and tones). 
This error detection equipment provides 
the system engineer with a real time 
awareness that a failure has occurred 
and allows him or her to properly safe 
the system in a timely manner. While 
this error detecting equipment makes the 
engineer aware of a subsystem failure, 
it does not (in most cases) give a cause 
of the failure. Due to the complexity 
of the Shuttle, both in terms of 
hardware and software, errors will fre- 
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quentiy arise during normal operations 
which are of an anomalous but harmless 
nature, but again due to system com- 
plexity, it is up to the systems en- 
gineer level of experience to differen- 
tiate the harmful from the benign er- 
rors. Frequently, an inexperienced (and 
even an experienced engineer) will en- 
counter a fundamentally benign failure 
yet diagnose it as a harmful one. In 
the interim, much paperwork is generated 
and a possible temporary interruption in 
launch processing may be experienced. 

It is at this point that a brief 
description of the current diagnostic 
methods should be discussed. As was 
stated earlier, the error detection 
equipment alerts the responsible system 
engineer that an anomaly has occurred, 
but not what has caused it. In order to 
ascertain the cause of an anomaly, the 
responsible system engineer (s) must ap- 
prehend what the overall system environ- 
ment was when the error occurred, and in 
order to do this, the engineer must rely 
on telemetry data. This telemetry data 
takes on two forms for DPS subsystems. 
The first is what is called dump data. 
Two of the DPS subsystems, the GPC' s and 
the MCDS have resident stored memory 
capacity (GPC of 104K and the MCDS of 
8K) . When an error occurs in either one 
of these two subsystems, the mal- 
functioning component is isolated and 
the stored memory is then transmitted 
down the LDB and placed on magnetic tape 
and paper printout . This memory content 
of the anomalous subsystem provides the 
engineer with an image or state of 
operation of the subsystem during the 
time at which the error occurred. This 
information is then combined with the 
second type of telemetry which is called 
downlist data. Downlist data is simply 
the encoded values, states and times of 
a large number of discrete and analog 
Shuttle parameters. It is from this raw 
data which the engineer (s) must review 
manually, that a diagnosis of the 
problem is obtained (hopefully) . Again, 
due to the complexity of the Shuttle, 
the amount of raw data is large (the 
MCDS alone has 8K of memory locations 
which must be reviewed manually) and, 
hence, is labor intensive and time con- 
suming . 

As stated in the abstract, we are 
describing an automated diagnostic sys- 
tem, the Multifunction CRT Display Sys- 
tem Diagnostic Tool (MDT) , which will 
aid in a more efficient processing of 
the DPS. Before going on to describe in 
more detail the functioning of the MDT, 
a brief description of the MCDS will be 
given. The MCDS is composed of three 
basic systems: (1) Keyboard, (2) Dis- 
play Electronic Unit (DEU), (3) CRT. 


The heart or the MCDS is the DEU which 
is the information processor for data 
between the CRT, keyboard and GPC' s and 
allows the astronauts to communicate to 
the GPC' s and vice versa. The DEU is 
composed of various logical circuitry 
for CRT data display, keyboard data and 
processing of GPC commands and data. In 
addition, the DEU has a memory store of 
8K in which to store data and commands 
for MCDS information processing, as well 
as, built-in test equipment (BITE) cir- 
cuitry. This BITE circuitry is composed 
of three 16 bit status registers and two 
mechanical flags. 

Now that a brief overview of the Shuttle 
processing and diagnostic environment 
has been described, a number of short- 
comings have been mentioned in reqard to 
the current nature of orbiter related 
diagnostics. These will now be stated 
more compactly: 

(1) As it stands now, current 
diagnostic techniques used to arrive at 
problem resolutions are labor intensive 
and time consuming. 

(2) Due to the complexity of the 
system (Shuttle, as a whole, and the 
DPS, in particular) being processed, the 
behavior of the system can, at times, 
behave in an unpredictable but benign 
manner with respect to normal opera- 
tions. This makes it difficult for the 
systems engineers to know whether an an- 
nunciated error is truly serious or 
trivial . 

Point 2 will have to be expanded upon, 
in order to make what follows in the 
rest of this paper consistent. The 
Shuttle has been in operation for almost 
11 years, hence, there is also a commen- 
surate 11 years worth of documented 
Shuttle behavior. The Shuttle behavior 
of most interest here is of the 
anomalous DPS kind, and this type of be- 
havior has been, in most cases, 
thoroughly documented. This documenta- 
tion will be described metaphorically as 
a triadic problem resolving knowledge 
base. The first part of this triad is 
known as a Problem Report (PR) database. 
This PR database is a paper system which 
is used to track the history and resolu- 
tion (if one exists) of launch process- 
ing related anomalies. The second part 
of the triad is a "user' s note" 
resource. The user note resource docu- 
ments and explains DPS subsystem be- 
havior which is anomalous but also 
benign with respect to the running of 
normal operations. The last and most 
important leg of this triad is the 
cerebral documentation which resides in 
the minds of our most experienced and 
astute engineers. It is from the inter- 
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Figure 1. 




action of this problem resolving 
knowledge base that solutions to our 
problems are derived. 

The inherent drawbacks of this system 
are as stated in (1) above, and also in 
that our triadic system must rely on a 
component (i.e. that astute systems en- 
gineer) who may not be accessible for 
some reason or another when a problem 
arises. These two weaknesses in the 
system, by their very nature, allow 
themselves to be alleviated to some ex- 
tent by automation. By incorporating a 
high-speed automated system, which has 
residing within it, both the paper his- 
tory of problems and the diagnostic wis- 
dom of our engineers (as best as that 
can be done) , an integrative tool can be 
added to our diagnostic triad to help 
support the system as a whole. For our 
particular purposes (DPS) , the MDT is a 
way of realizing an automated diagnostic 
system which can aid us in doing busi- 
ness with misbehaving avionics boxes. 
This system is to fulfill the following 
four goals: 

(1) Monitoring of downlist data 
for MCDS anomalies 

(2) Testing the downlist data for 
the presence of pre-defined error condi- 
tions 

(3) Presentation to the user of 
probable cause of the failure 

(4) Presentation of problem report 
and user note information corresponding 
to the failure detected. 


How the MDT proposes to attain these 
four goals is the topic of our next sec- 
tion . 

The MDT is contracted to Rockwell Inter- 
national Corporation, Launch Support 
Services. The development team consists 
of Rockwell test personnel from the 
Avionics Software organization at KSC, 
Florida, and Artificial Intelligence 
personnel from the Expert Systems Ap- 
plications organization at Downey, 
California. In addition. Abacus 
Programming Corporation personnel were 
added to augment the team. 

The host computing system will be a SUN 
SPARCstation 1 Plus. This system was 
selected for its ability to perform 
multi-tasking with its UNIX based 
operating system. In addition, the SUN 
SPARCstation was selected to maintain 
compatibility between this advisory sys- 
tem and future firing room applications. 
The "C" Language Integrated Production 
System (CLIPS) was selected as the ex- 
pert system shell to be utilized by the 
MDT. CLIPS is a forward chaining rule- 
based language that provides an in- 
ference engine and a language syntax 
that lends itself to interfacing with 
externally defined functions. As more 
and more firing room applications are 
automated, it is believed that a stan- 
dard shell with increased capabilities 
will be available with the cost being 
shared among projects. 

The conceptual system (Figure 1) will be 
powered on in the firing room at all 
times when the orbiter is powered on. 


333 









Figure 2. 



The system will be in a "standby" mode 
of operation awaiting occurrence of 
off-nominal conditions . While in 

"standby" mode, test engineering person- 
nel may pictorially view the current 
configuration and status of the MCDS 
system as it is on-board the orbiter. 
The MDT will dynamically update the Sys- 
tem State Model display and monitor for 
errors with near real time data from a 
telemetry link to the firing room Common 
Data Buffer (CDBFR) . 

The process of acquiring the telemetry 
data (Figure 2) is a most challenging 
process, since the present firing room 
hardware is not compatible with the SUN. 
A method of acquiring telemetry data has 
been developed by the Advisory System 
Data Acquisition Project at KSC [3] . 
This telemetry data will be read from 
the Launch Processing System (LPS) com- 
mon data buffer via a data control 
program. This program scans the buffer 
once a second and sends the data to a 
VME subsystem that blocks the incoming 
data stream into Ethernet frames and 
sends it out on an Ethernet line to 
various system users. The MDT com- 
munications process accepts the incoming 
data and places it in a data buffer 
residing on the SUN. This buffer is in 
shared memory and is accessible to the 
various applications resident on the 
SUN. 


The incoming data will be monitored for 
off-nominal conditions. More ex- 
plicitly, the three status registers for 
each of the four DEU' s will be monitored 
for any abnormal bit pattern change. 
The monitoring of these 12 parameters 
can detect up to 90% of all MCDS 
failures. If an MCDS failure that is 
not triggered by status register changes 
occurs, the engineer will be able to 
utilize a manual mode of operations in 
which an analysis can be performed 
without being initiated by changes in 
telemetry data. 

Once an anomalous condition is recog- 
nized, a "snapshot" of the MCDS environ- 
ment is taken and saved for the system 
to begin the error analysis process. 
This snapshot process allows the system 
to complete the analysis process of c 
single error without possible loss of a 
secondary error. Once an error is 
detected, a set of pre-defined error 
conditions are checked. Each error con- 
dition requires the analysis of a unique 
set of data from the buffer and possibly 
the user. The user will be queried in 
situations where telemetry data is 
either unavailable or insufficient. In 
general, user requests will be data that 
may only be obtained by visual inspec- 
tion of the MCDS (i.e. blank CRT's or 
tripped mechanical flags) . The need for 
user supplied data will be kept to a 
minimum to enhance the automated nature 
of the system. 
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The system will evaluate and eliminate 
possible causes of the given failure 
condition. A hierarchy will exist among 
the pre-defined error conditions such 
that if an error condition of high prob- 
ability is determined to be the cause, 
other unlikely conditions will not be 
checked. The possibility does exist for 
more than one probable cause of a 
failure to be displayed to the user. 
This could occur if telemetry data is 
temporarily unavailable or if the user 
did not supply the necessary data. As 
always, the systems engineer makes the 
ultimate decision concerning the most 
probable cause of the failure using data 
obtained from the MDT . The data used in 
the error analysis process will be saved 
to a file for later use in the re- 
creation of failure scenarios for either 
re-evaluation or training. The data 
necessary to perform the analysis 
process is being provided by Rockwell 
and NASA KSC employees . For each of the 
possible failure conditions currently 
recognized, a thorough review of his- 
torical data, both documented and un- 
documented, must be performed. This 
data is then organized into "rules” that 
will be encoded into the system by Rock- 
we 11, Downey . 

All results of the error analysis will 
be displayed to the CRT and output to a 
printer. The results are the coordina- 
tion of the system environment at the 
time of the error, probable causes of 
the failure, possible troubleshooting 
steps to be taken, and references to 
past problems and user notes pertaining 
to the failure condition. The coordina- 
tion of this information is currently 
done manually by the systems engineers 
and is a time consuming process. All 
the information obtained from the 
results will be utilized to support 
closure of paperwork generated at KSC 
due to the MCDS failure. 

The mode of operations described thus 
far constitutes the "Automatic" mode of 
operation. This mode will have the 
highest priority and will automatically 
be run as a foreground task upon receipt 
of an anomalous condition. This system 
also includes "Manual" and "Replay" 
modes that are available on an as-needed 
basis. The selection of either mode 
presents the user with sub menu' s to ac- 
cess the MDT functions. The user will 
have the ability to perform "what-if" 
analysis on the MCDS in a test environ- 
ment, replay already analyzed failures, 

review past PR and user note databases, 
and review the results of past failure 
analysis . 


The MDT is viewed by the systems en- 
gineers at KSC as a highly desirable 
concept. By automating the processes 
performed manually at this time, MCDS 
failure recognition and resolution can 
be performed more rapidly and effi- 
ciently. This system will also provide 
invaluable training experience for sys- 
tems engineers. The MDT system require- 
ments and specifications, as defined, 
are such that the previously mentioned 
diagnostic triad representing an 
automated problem resolving knowledge 
base can be utilized. 

The MDT is a 3 year project in which 
the first year prototype will con- 
centrate on the development of a proof- 
of-concept prototype to demonstrate the 
four goals defined. The prototype will 
encompass the utilization of telemetry 
downlist data to perform diagnostics of 
the MCDS. The prototype to be delivered 
by October 1990 will perform automated 
analysis of between 5 and 10 errors. 
Its manual mode will provide the 
capability to search the PR and user 
note databases and to retrieve informa- 
tion about DEU status register bits. 
The second year will include the addi- 
tion of the remaining known error 
scenarios, as well as, the addition of 
simulated DEU dump data. The dump data 
will enhance the diagnostic capabilities 
of the MDT. The final year will consist 
of integration with the Expert System 
for Operations Distributed Users System 
(EXODUS) [2] and the capture of near 
real time DEU dump data. This should 
complete the firing room implementation 
of the MDT. 

The developers of the MDT view this con- 
cept as a pioneering diagnostic system 
for all DPS subsystems (GPC, MDM, MMU) . 
A long term goal of the MDT project is 
that an eventual integration of all DPS 
diagnostics will be realized in a dis- 
tributed diagnostic system. Such a dis- 
tributed knowledge base concept for 
launch processing is already being in- 
vestigated at KSC under the EXODUS 
program. It is hoped that the knowledge 
gained with the MDT and other Shuttle 
advisory systems currently being 
developed [1] will aid the EXODUS 
program and, in turn, help in the 
realization of a distributed DPS diag- 
nostic system. 
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APPENDIX 

Acronyms 

BITE - Built In Test Equipment 
CDBFR - Common Data Buffer 

CLIPS - "C" Language Integrated Produc- 
tion System 
CRT - Cathode Ray Tube 
DEU - Display Electronics Unit 
DPS - Data Processing System 
EXODUS - Expert System for Operations 
Distributed Users System 
GPC - General Purpose Computer 
LDB - Launch Data Bus 
LPS - Launch Processing System 
MCDS - Multi-Function CRT Display System 
MDM - Multiplexer Demultiplexer 
MDT - MCDS Diagnostic Tool 
MMU - Mass Memory Unit 
PR - Problem Report 
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